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The  Warfighter  Associate:  Objective  and  Automated 
Performance  Metrics  for  Mission  Command 

Norbou  Buchler^,  Laura  Marusich^,  Jonathan  Z.  Bakdash^,  Stacey  SokolofP,  Robert  Hamm^ 
1.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD,  USA 
2.  Veloxiti,  Inc.,  Alpharetta,  GA 

It  has  been  assumed  that  within  Mission  Command  and  Network  Enabled  Operations 
information  superiority  results  in  greater  information  sharing,  situational  awareness, 
collaborative  decision-making,  and  agility;  all  of  which  ultimately  contribute  to  greater 
mission  effectiveness.  However,  demonstrating  the  truth  of  these  assumptions  requires 
objectively  measuring  individual  and  collaborative  Soldier  performance  and  information 
sharing  in  networked  environments,  which  is  an  enduring  challenge.  The  Warfighter 
Associate  (WA)  is  an  Intelligent  Agent  software  system  that  uses  doctrinally-based  knowledge 
engineering  to  offer  decision-support  for  Soldier  cognitive  workflows.  The  underlying 
knowledge  representation  can  serve  as  state  traces,  measuring  aspects  of  Soldier  cognitive 
performance  across  scenario  runtime  as  dynamic  events  unfold  on  the  battlefield.  This 
approach  can  unobtrusively  and  automatically  capture  aspects  of  Mission  Command 
performance  over  time.  Specifically,  the  WA  currently  captures  the  following  performance 
metrics: 

1)  Objective  cognitive  workload  over  time 

2)  Responsiveness  to  events  in  dynamic  environments 

3)  Course  of  action 

4)  Staff  collaborations  and  manifest  agility 

Our  approach  captures  Soldier's  goals  and  the  plans  to  achieve  those  goals  in  real-time  as  the  work 
dynamically  unfolds  as  a  complex  network  of  cognitive  tasks.  The  capability  of  an  analyst  to  record, 
view,  and  analyze  work  activity  dynamically  and  unobtrusively  over  specified  epochs  of  time  is,  to  our 
knowledge,  unprecedented. 


1.  INTRODUCTION 

Military  operations  are  inherently  complex  human  endeavors.  Army  commanders  and  their  staff 
collectively  face  difficult  and  stressful  challenges  in  managing  battlefield  operations.  Warfare  is  chaotic 
and  incredibly  complicated,  and  resolving  the  attendant  ambiguity  of  the  battlefield  is  a  cognitive 
challenge  of  the  first  order.  At  critical  times.  Soldiers  are  under  tremendous  pressure  to  quickly  analyze 
overwhelming  amounts  of  incomplete  and  sometimes  contradictory  data  and  to  make  decisions  that 
have  immediate  impacts  to  mission  success  and  human  life.  The  consequences  of  those  actions  are  not 
always  intuitive  or  predictable. 

The  increased  complexity  of  today's  operational  environment  presents  additional  challenges.  Advances 
in  networking  technology  have  transformed  the  way  that  Mission  Command  networks  function.  Mission 
Command  operations  involve  large,  interacting,  and  layered  networks  of  staff  personnel  communicating 
across  echelons  of  command.  Operating  in  such  a  broadly  collaborative  and  information-rich 
environment  has  the  potential  to  confer  unprecedented  advantages  in  directing  and  responding  to 
battlefield  events  (National  Research  Council,  2005);  however,  it  can  also  present  a  number  of 
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challenges.  Current  operational  environments  are  challenged  by  the  shift  to  multifaceted  civil-military 
operations  with  joint,  interagency,  and  multinational  partners  and  the  overabundance  of  information 
available  across  Army  networks.  In  spite  of  Mission  Command's  profound  dependence  on  networks, 
fundamental  knowledge  about  the  conduct  of  network-enabled  communication  operations  is  lacking, 
especially  at  the  cognitive  and  social  levels.  The  dependence  of  the  Mission  Command  staff  on  robust 
interaction  of  networks  is  clear  from  its  transformation  to  network-enabled  operations  (Alberts  & 
Garstka,  2004;  National  Research  Council,  2005). 

The  Challenge  of  Network-enabled  Operations 

The  transformation  of  U.S.  and  NATO  countries  to  network-enabled  operations  (NEO)  has  proceeded 
under  a  conceptual  framework  of  network-centric  warfare  with  four  main  tenets  (Alberts  &  Garstka, 
2004): 

•  A  robustly  networked  force  improves  information  sharing  and  collaboration 

•  Such  sharing  and  collaboration  enhance  the  quality  of  information  and  shared  situational 
awareness 

•  This  enhancement,  in  turn,  enables  further  self-synchronization  and  improves  the  sustainability 
and  speed  of  command 

•  The  combination  dramatically  increases  mission  effectiveness 

These  four  tenets  capture  the  current  challenges  and  goals  of  the  Army  transformation.  However,  they 
are  in  need  of  systematic  study.  For  instance,  it  may  be  the  case  that  information  sharing  in  NEO  has 
increased  the  quantity  but  not  the  quality  of  information.  Indeed,  today's  battlefield  commanders  and 
staff  are  inundated  with  huge  amounts  of  information/intelligence  that  are  transmitted  and  stored 
across  increasingly  sophisticated  Army  networks.  In  response  to  the  complexity  of  the  battlefield 
environment,  the  current  impulse  and  design  of  technical  systems  is  to  gather  and  process  more  and 
more  information  (Lynch,  2008).  One  concern  is  that  such  a  data  rich  environment  can  quickly 
overwhelm  and  paralyze  human  decision-making  capacities.  Another  concern  is  that  human  cognition  is 
being  reduced  to  managing  complex  computer  databases  and  configuring  displays,  at  the  expense  of 
engaging  higher-ordered  human  faculties  of  critical  thinking,  sense  making,  and  reasoning  about  the 
battle-space.  A  Vietnam-era  commanding  officer,  COL  Ted  FichtI  (retired)  who  recently  observed  Mission 
Command  at  a  Division  and  Brigade  level  US/UK  coalition  exercise,  turned  aside  and  remarked,  "My 
God!  They've  taken  the  thinking  out  of  Mission  Command!"  (FichtI,  2010). 

Managing  the  convergence  of  people,  information,  and  technology— a  soclotechnical  system  (Walker,  et 
al.,  2009)  —  is  a  defining  challenge  of  our  era.  Despite  the  best  intentions  of  solution  providers  there  are 
inevitably  situations  where  systems'  capabilities  are  not  in  complete  alignment  with  Soldiers' 
information  needs.  The  inherent  complexity  and  dynamic  nature  of  military  operations  entails  an 
unbounded  problem  space:  all  possible  intricacies  involved  in  the  dynamics  of  the  mission  command 
space  and  the  interrelationships  of  technical  systems  are  difficult  to  predict  in  advance. 

Objectively  measuring  Mission  Command  effectiveness  is  a  formidable  Army  challenge.  Researchers 
have  difficulty  capturing  the  intricacies  of  individual  and  team  cognition,  particularly  as  Soldiers  confront 
environmental  challenges  with  multiple  overlapping  problem  spaces.  How  to  develop  predictive 
performance-based  models  of  such  emergent  problem  solving  behaviors  is  an  open  problem  in  need  of 
solutions.  One  such  solution  is  automated  collection  and  aggregation  of  empirical  data  derived  from 
field  or  laboratory  studies.  An  important  research  objective  is  to  develop  the  scientific  experimental 
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infrastructure  and  measures  to  capture  Mission  Command  effectiveness,  focused  on  theory 
development,  experimentation,  hypothesis  testing,  and  data  collection  relevant  to  operations.^ 

An  Experimental  Infrastructure  for  Mission  Command 

In  this  paper,  we  demonstrate  a  novel  approach  for  the  evaluation  of  Mission  Command  performance 
using  automated  real-time,  network-based  measures  of  team  cognition  and  team  effectiveness.  Our 
approach  uses  the  state  trace  of  activation  of  a  prototype  intelligent  software  agent  technology—  the 
Warfighter  Associate—  to  model  specific  user  workflows  in  response  to  unfolding  events  on  the 
battlefield  over  the  course  of  a  military  scenario. 

The  Warfighter  Associate  (WA)  is  an  intelligent  decision  support  tool  under  development  for  Mission 
Command  that  considers  user  intent,  state  of  the  world,  and  domain  specific  knowledge  to  recommend 
a  course  of  action.  The  WA  seeks  to  improve  Mission  Command  performance  by  shortening  the  cycle 
time  from  data  gathering  to  decisions.  It  uses  a  doctrinally-based  knowledge  representation  to  model 
role-specific  workflows  and  continuously  monitors  the  state  of  the  operational  environment  to  enable 
decision-support,  delivering  the  right  information  to  the  right  person  at  the  right  time. 

We  tested  the  WA  over  the  course  of  a  military  scenario,  demonstrating  how  unfolding  events  on  the 
battlefield  activate  WA  mission  concepts,  user  goals,  and  the  plans  to  achieve  them.  Capturing  the  state 
trace  of  WA  knowledge  activation  over  scenario  runtime  constitutes  an  important  research  breakthrough 
in  that  it  contributes  to  the  development  of  objective  metrics  for  assessing  Command  and  Control  (C2) 
effectiveness.  The  most  recent  version  of  the  WA  includes  an  analysis  tool  to  visualize  mission  concepts, 
tasks,  and  task  performance  in  real-time.  Based  on  novel  quantitative  metrics  derived  from  the  state 
traces  of  knowledge  activation  in  the  WA,  we  can  operationally  define  important  cognitive  and  Army 
concepts  such  as  cognitive  workload,  multi-tasking,  task  completion  timing,  force  synchronization, 
information  sharing,  and  decision  quality.  This  constitutes  a  potentially  important  methodological 
breakthrough  in  capturing  objective  group  performance.  Metrics  are  essential  to  developing,  evaluating, 
and  improving  soldier-system  interfaces  and  performance;  and  in  our  case,  can  also  be  collected 
unobtrusively.  In  sum,  we  describe  an  associate  system  that  supports  the  full  sequence  of  "data  to 
decisions"  to  ensure  that  it  occurs  in  a  timely  and  accurate  manner,  and  provides  a  novel  class  of  metrics 
to  assess  the  operational  efficiency  of  mission  command. 

II.  OVERVIEW  -  THE  WARFIGHTER  ASSOCIATE  SYSTEM 

The  Warfighter  Associate  is  an  example  of  a  knowledge-based  system,  an  intelligent  agent  technology 
that  uses  collective  knowledge  from  military  doctrine,  subject  matter  experts,  and  other  information 
sources  (see  Arkerkar  &  Sajja,  2011).  Compared  to  other  types  of  Al  technology,  knowledge-based 
Associate  Systems  offer  a  high  degree  of  flexibility,  rationale  for  recommendations  rooted  in  knowledge 
and  available  information,  and  a  proactive  and  reactive  manner  of  operation.  Ultimately,  these  three 
characteristics  are  necessary  for  an  Al  system  to  be  responsive  to  the  uncertainty  and  complexity  of 
information  on  today's  battlefield  and  systematically  tailored  to  staff  workflows  and  the  unit's  standard 
operating  procedures. 

To  make  timely  and  optimal  recommendations,  the  WA  uses  collective  knowledge  from  doctrine  and 


^  This  statement  reflects  the  out-brief  recommendation  by  Working  Group  #4  (Analysis  of  Human  Decision  Making  in 
Networked  Environments)  in  a  workshop  by  the  Military  Operational  Research  Society  (MORS)  entitled  "Joint  Framework  for 
Measuring  C2  Effectiveness"  held  at  the  Johns  Hopkins  University  Applied  Physics  Laboratory  in  Laurel,  MD  (23-26  January, 
2012).  http://www.mors.org/events/2012c2.aspx 
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subject  matter  experts  to  infer  human  intention  (goals)  and  reason  about  the  best  manner  to  achieve 
them  (plans)  given  the  state  of  the  world  (operational  environment  monitor).The  WA  is  flexible;  it  is  a 
plug-in  capability,  and  the  WA  knowledge  is  independent  of  the  system  it  is  integrated  with.  Thus,  the 
system  can  run  as  a  stand-alone  application,  or  can  be  lightly  and  seamlessly  integrated  with  current  or 
future  systems. 

The  WA  has  a  minimal  level  of  automation  (see  levels  of  automation  by  Parasuraman,  Sheridan,  & 
Wickens,  2000),  suggesting  a  set  of  specific  courses  of  action  but  ultimately  the  decisions  and  actions  are 
up  to  the  Soldier.  Rationale  for  the  suggestions  is  provided  to  the  Soldiers,  for  instance:  the  closest  and 
best  available  intelligence,  surveillance,  and  reconnaissance  (ISR)  resources  in  responding  to  a  battlefield 
event  based  on  a  combination  of  availability,  distance,  speed,  and  capabilities.  Transparency  is  a  key 
system  design  if  recommendations  are  to  be  accepted,  selected,  and  acted  upon  by  human  operators 
(Parasuraman  &  Riley,  1997). 

An  Associate  System  is  software  driven  by  domain  knowledge  that  is  designed  to  assist  a  human 
operator.  It  can  enable  the  Mission  Command  staff  to  engage  in  a  high  operations  tempo  by 
understanding  their  own  decision  cycle  and  reacting  or  pre-acting  to  their  goals  and  by  gaining  possible 
insights  into  higher  order  effects  and  unintended  consequences.  A  relevant  paradigm  for  Mission 
Command  is  the  late  Colonel  John  Boyd's  concept  of  the  "OODA  Loop"  (Boyd,  1986;  see  also  Polk,  1999) 
where  OODA  stands  for  Observe-Orient-Decide-Act.  The  OODA  loop  defines  a  process  by  which  an 
individual  or  team  responds  to  a  situation,  with  the  need  to  repeatedly  make  decisions  in  light  of  dynamic 
events,  and  is  closely  related  with  the  concept  of  Situation  Awareness  as  developed  by  Mica  Endsley  (see 
Endsley  &  Garland,  2009).  For  our  purpose  of  supporting  decision-making,  Boyd's  theory  is  central  in 
defining  the  essential  cognitive  cycle  and  collaborative  dimensions  of  Mission  Command  performance, 
particularly  in  relation  to  an  adversary. 

The  Mission  Command  work  domain  is  an  obvious  candidate  for  computer  support.  The  demands  to 
human  information  processing,  which  involve  observing  and  monitoring  communication  streams,  as  well 
as  organizing,  combining,  and  evaluating  data  and  intelligence,  can  quickly  overwhelm  cognitive 
capabilities.  The  Associate  can  filter  through  vast  amounts  of  data  looking  for  information  of  importance 
to  the  user  based  on  the  user's  intent.  It  does  so  by  making  abstract  and  aggregate  conclusions  about  the 
state  of  the  world  in  a  more  automated  fashion,  which  in  humans  normally  requires  both  attention  and 
expertise.  As  information  is  being  assessed,  the  intelligent  agent  can  help  in  the  development  of  basic  and 
advanced  situation  awareness,  including  the  identification  of  patterns,  correlation  of  different  data, 
diagnoses,  problem  solving,  and  even  goal  setting.  It  can  present  to  the  user  the  "best"  solution  to  this 
problem  based  on  currently  available  information,  but  support  the  user  if  a  different  course  of  action  is 
chosen.  In  the  action  part  of  the  cycle,  the  associate  may  be  authorized  to  perform  many  of  the  routine 
tasks  that  could  distract  the  user  from  the  important  events  occurring.  An  Associate  System  also  observes 
the  actions  undertaken  by  a  human  operator,  combining  those  actions  with  the  state  of  the  world  to 
determine  the  operator's  current  objectives  and  activities.  Based  on  the  assessment  of  the  state  of  the 
world  and  the  activities  and  objectives  of  the  human  operator,  the  system  can,  within  the  bounds  of  its 
authority,  carry  out  activities  on  behalf  of  the  user,  make  the  user  aware  of  events  particularly  relevant  to 
his  activities,  recommend  courses  of  action,  and  manage  the  information  content  of  the  user's  displays. 

The  Warfighter  Associate  (WA)  is  intended  to  be  a  smart  and  seasoned  assistant  to  the  human  user, 
designed  to  follow  the  human's  lead,  aiding  whenever  necessary  without  the  need  for  explicit 
instructions  and  within  its  bounded  discretion.  The  human  user  preserves  all  opportunities  to  interact 
normally  and  perform  all  system  tasks  completely  manually.  The  goal  of  the  Associate  System  is  to  foster 
the  functional  integration  of  the  sociotechnical  Soldier-system. 
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III.  WARFIGHTER  ASSOCIATE  ARCHITECTURE 


The  WA  is  a  software  system  that  models  tasks  and  task  performance  in  complex  real-time  operational 
environments.  The  WA  can  be  used  both  as  a  decision  aiding  system  and  as  a  normative  model  of  task 
selection  and  performance  by  human  users.  It  is  composed  of  a  set  of  functions  that  create  a  closed  loop 
cognitive  engine  that  uses  explicitly  declared  knowledge  bases  to  perform  situation  assessment, 
dynamic  planning,  action  execution  and  coordination  across  multiple  actors. 

The  Warfighter  Associate  cognitive  model  has  the  following  components: 

•  Domain-specific  knowledge 

•  The  ability  to  accept  situational  data  as  input  data 

•  The  ability  to  accept  user  actions  as  input  data 

•  Algorithms  to  assess  user  actions,  situational  data,  in  accordance  with  the  domain  knowledge 
and  do  one  or  more  of  the  following:  provide  notifications,  provide  suggestions,  or  perform 
system  actions 

As  a  brief  example,  given  a  reported  improvised  explosive  device  (lED)  event,  the  associate  system 
provides  to  the  user—  here,  an  intelligence  officer  (S2)— a  recommended  list  of  nearby  surveillance 
assets  that  could  be  repositioned  to  quickly  provide  eyes-on  the  event  location.  The  associate  system's 
knowledge  representation  'knows'  that  in  response  to  an  lED  event  this  user  will  likely  want  to 
reposition  known  nearby  surveillance  assets  with  particular  known  capabilities.  The  term  'known' 
indicates  that  the  associate  system  is  aware  of  the  current  state  of  the  world  such  as  the  position  and 
capabilities  of  the  available  surveillance  assets.  In  this  manner,  the  associate  system  is  proactive  in 
aligning  the  likely  goals  and  decision-making  authority  of  the  warfighter  with  the  means  to  achieve 
them. 

A  key  to  understanding  the  mechanics  of  the  associate  systems  is  to  think  of  the  gears  of  the  process  as 
a  user  intent  interpretation  engine.  Observed  actions  drive  the  intent  interpretation  capability  of  the 
Associate.  By  observing  the  Warfighter's  actions,  the  Associate  can  interpret  which  scripts  the 
warfighter  is  following,  allowing  it  to  refine  its  user  intent  representation—  the  set  of  active  plans  and 
goals—  so  that  it  can  support  the  warfighter  based  upon  his  actual  intent,  even  if  that  intent  differs  from 
the  associate's  recommendations.  Specifically,  the  capability  to  model  the  knowledge  used  to  represent 
the  decision  making  process  and  to  drive  the  behavior  of  the  Warfighter  Associate  is  provided  by  the 
Velox’  Intelligent  Software  Suite.  Figure  1  is  a  diagram  of  Velox*. 

There  are  two  interrelated  knowledge  representations  in  the  Warfighter  Associate:  the  Observe-Orient 
(0-0)  graph,  the  Decide-Act  (D-A)  graph,  and  scripted  plans  that  encapsulate  procedural  knowledge.  The 
first  of  the  two  main  distinct  knowledge  structures  is  the  Observe-Orient  (0-0)  graph.  The  0-0  graph 
represents  general  and  situational  knowledge  about  the  current  world-state  ("who,"  "what,"  "when," 
and  "how-much"),  and  linkages  among  beliefs  (concepts)  represented.  The  0-0  graph  provides  a  means 
for  distinguishing  between  beliefs  about  the  state  of  the  environment  and  its  true  state.  Beliefs  are 
dynamically  updated  as  a  result  of  observations  in  the  form  of  incoming  data  about  the  perceived  state 
of  some  aspect  of  the  environment;  the  updated  situation  awareness  influences  the  Associate's 
purposeful  interactions  with  the  environment  through  the  0-0  graph.  The  links  between  the  beliefs 
contain  instructions  for  computing  higher  level  aggregations  and  abstractions  contained  in  each  "child" 
belief  from  the  data  contained  within  each  "parent"  belief.  Uncertainty  calculations  may  also  be 
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contained  in  the  links,  allowing  different  sources  of  information  to  receive  more  or  less  influence  in 
shaping  a  belief  value. 

The  second  knowledge  structure  is  the  Decide-Act  (D-A)  graph,  which  is  a  graphical  depiction  of  the 
hierarchical  task  structure  in  the  system  being  modeled.  This  structure  allows  a  principled  separation 
between  the  intended  future  state  (goal)  from  the  means  through  which  that  goal  might  be  attained 
(plan).  It  supports  plan  generation  and  plan  recognition  in  dynamic,  uncertain  environments.  A 
collection  of  goals  and  possible  plans  for  achieving  each  goal  makes  up  a  course  of  action  for  a  group. 
The  D-A  graph  is  defined  to  represent  the  alternative  ways  that  goals  can  be  achieved,  so  each  plan  child 
of  a  goal  is  a  possible  means  to  achieve  the  parent  goal.  Plan  nodes  are  then  decomposed  into  sub 
goals,  with  decomposition  in  this  manner  continuing  until  the  level  of  basic  interactions  is  reached  in  the 


Figure  1.  Velox®  IntelligentSoftware  Suite  with  Observe-Orient  and  Decide-Act  Graphs 

form  of  scripts.  AD-A  graph  may  provide  for  the  intentions  of  many  types  of  groups  within  a  model.  By 
observing  the  human's  actions,  and  interpreting  them  in  the  context  of  the  task  models  in  the  D-A 
graph,  the  Warfighter  Associate  can  infer  the  human's  intentions  by  explaining  them  in  terms  of  implied 
plans  for  achieving  shared  task  goals. 


To  provide  dynamic  behavior  to  the  model,  nodes  (Beliefs,  Plans,  and  Goals)  have  dynamic  life  cycle 
states.  In  the  Observe-Orient  (0-0)  graph,  life  cycle  state  represents  the  prominence  of  a  belief,  allowing 
beliefs  that  are  no  longer  supported  by  evidence  to  become  forgotten.  The  feasibility  and  desirability  of 
any  particular  course  of  action  (path  through  the  Decide-Act  graph)  may  depend  on  what  the  group 
believes  about  the  environment  and  what  stage  of  development  the  group  is  in  (0-0  graph).  To 
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implement  the  dynamic  connection  between  the  state  of  the  environment  and  the  possible  courses  of 
action,  the  model  framework  provides  a  belief  monitoring  mechanism.  For  example,  a  high-level  plan 
"to  survive"  may  subscribe  to  messages  about  threats.  If  an  lED  is  reported,  the  subscription  monitor 
will  fire,  triggering  planning.  The  group  may  consider  multiple  ways  to  accomplish  a  goal  to  diffuse  the 
lED.  The  specific  way  selected  will  activate  monitors  for  the  relevant  information  in  the  0-0  graph.  For 
example,  a  plan  to  use  the  Z"'*  Explosive  Ordinance  disposal  platoon  will  monitor  the  0-0  graph  for 
obstacles  on  the  route  between  the  platoon's  location  and  the  location  of  the  lED.  If  a  conditional 
statement  in  a  monitor  is  found  to  be  true,  the  detected  event  may  be  used  to  transition  a  plan  or  goal 
to  a  different  life  cycle  state.  In  this  way,  plans  and  goals  that  are  no  longer  feasible  or  desirable  can  be 
replaced  with  more  desirable  ones  as  the  state  of  the  beliefs  change  over  time. 

The  Warfighter  Associate  decomposes  high-level,  abstract  plans  into  lower-level,  more  concrete  plans. 
Eventually,  all  decisions  are  made,  and  a  plan  can  then  be  executed  by  running  a  script.  Scripts  contain 
actions  and  logic  to  determine  when  each  action  is  appropriate.  Actions,  which  are  manipulations  of  the 
world  state,  can  be  performed  by  the  Associate  ("performed  action")  or  executed  by  the  human  and 
observed  by  the  Associate  ("observed  action").  Since  Associates  are  mixed-initiative,  whether  an  action 
is  performed  or  observed  can  be  determined  during  runtime.  Examples  of  actions  include  calling  a  route 
planner,  querying  a  database,  or  re-tasking  an  asset.  As  a  key  part  of  the  OODA  loop,  actions,  by  their 
definition,  change  the  state  of  the  world,  which  causes  the  0-0  graph  to  be  updated.  This  may  result  in 
re-planning,  which  may  then  cause  additional  actions  to  be  performed. 

IV.  WARFIGHTER  ASSOCIATE  KNOWLEDGE 

Knowledge-engineering  tools  are  used  in  developing  and  maintaining  the  0-0  and  D-A  graphs  and  the 
associated  plan  scripts.  These  tools  enable  doctrinal  and  Subject  Matter  Expert  knowledge  to  be  easily 
entered  and  put  into  a  format  where  it  can  drive  Warfighter  Associate  behavior.  The  knowledge  comes 
from  approximately  twenty  different  Army  publications,  as  well  as  subject  matter  expertise  in  response 
to  high-intensity  events  such  as  lED  detonations,  medical  evacuation  (MEDEVAC)  operations,  high  value 
target  (FIVT)  sightings.  Restricted  Operating  Zone  (ROZ)  establishment,  and  so  on. 

As  currently  developed,  the  Warfighter  Associate  is  able  to  respond  to: 

•  All  normal  doctrinal  operations 

•  TIC 

•  Personnel  recovery  /  downed  aircraft 

•  MEDEVAC  and  casualty  evacuation  (CASEVAC) 

•  lEDs,  including  vehicle-borne  (VBIED),  suicide-vest  (SVIED),  and  potential  (P-IED) 

•  Indirect  fires  (e.g.  mortar  attack)  with  point-of-origin  (POO)  and  point-of-impact  (POI) 

•  Restricted  operations  zone  (ROZ)  for  airspace  de-confliction 

•  Fligh  value  targets  (FIVT) 

•  Intelligence  Surveillance  Reconnaissance  (ISR)  asset  management 

•  Unit  boundary  coordination 

•  Joint  and  coalition  coordination 

•  Minefields 

•  Civilian  demonstrations 

•  Air  threats 

•  Small  Arms  Fire 
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The  Knowledge  Engineering  tools  that  are  part  of  the  Warfighter  Associate  enable  doctrine  or  subject 
matter  expertise  to  be  added  or  modified  within  the  knowledge  base  without  requiring  any  changes  to 
the  underlying  software  or  system  architecture.  The  0-0  and  D-A  graphs  sit  on  top  of  the  associate 
architecture  and  thus  are  readily  modifiable,  for  instance  to  tailor  the  knowledge  to  a  Commander's 
standard  operating  procedures  or  to  evolving  threats  on  the  battlefield. 

The  Warfighter  Associate  provides  a  dynamic  real-time  model  of  human  intention  and  provides  mission 
command  with  the  ability  to  engage  in  high  up-tempo  operations  with  support  for  planning  and  plan 
execution  and  fault  tolerance.  It  fosters  rich  human-computer  collaboration  and  addresses  the 
complexity  of  mission  command  in  a  network-enabled  operational  environment. 

V.  METRICS  FOR  MISSION  COMMAND 

The  underlying  activation  of  the  knowledge  structures  in  the  Warfighter  Associate  can  provide  a 
dynamic  real-time  model  of  human  intention  that  has  the  potential  to  support  the  analytical  community 
with  novel  classes  of  metrics.  This  addresses  the  long-standing  problem  of  objectively  measuring  mission 
command  effectiveness  at  the  individual  and  team  level.  These  metrics  can  be  derived  from  the  pattern 
of  node  activations  from  the  knowledge-base  (0-0  and  D-A  graphs)  and  used  to  gauge  the  effectiveness 
of  the  mission  command  staff. 

Examples  include: 

•  Cognitive  workload  (number  of  concurrently  active  goals  across  time) 

•  Currently  active  plans  &  goals 

•  Timing  to  complete  tasks 

•  Necessary  collaborations  (shared  plans  &  goals) 

•  Force  synchronization  (timeliness  of  distributed  sub-goal  satisfaction) 

Since  the  knowledge  includes  a  dynamic  model  of  the  operational  environment,  and  the  user  interface  is 
updated  with  appropriate  alerts,  suggestions,  and  highlighting  of  critical  information  as  time  progresses, 
the  knowledge  representations  serve  as  state  traces.  The  state  traces  depict  the  cognitive  work 
demands  on  the  mission  command  staff  across  scenario  runtimes  as  dynamic  events  unfold.  A  suite  of 
data  analysis  tools  (DAT)  was  developed  to  look  at  the  underlying  node  activations  in  the  knowledge 
structures,  which  include  an  instance  viewer,  a  log  analyzer,  and  a  shared  event  analyzer. 

Data  Analysis  Toolset 

The  DAT  has  four  primary  components  that  are  used  to  view  node  activations  and  a  log  output  of 
moment-by-moment  changes  in  activation  states: 

1)  Heat  Map:  Active  nodes  in  the  knowledge  structure  which  provides  a  heat-map  for  cognitive 
work  analysis 

2)  Cognitive  Workload  Chart:  Time-series  figure  of  activation  and  deactivation  of  nodes  which 
shows  cognitive  workload 

3)  Reports  View  and  Task  Completion  Timing:  The  start  and  stop  times  for  events,  e.g.,  lED,  TIC. 
Events  include  input  data,  alerts,  recommended  COAs,  and  actual  COAs  and  are  anything  that 
triggers  node  activations  and  deactivations  in  the  knowledge  structures. 

4)  Shared  Event  Analyzer:  Events  can  be  shared  across  roles,  requiring  collaboration.  For  example, 
an  air  MEDEVAC  requires  coordination  between  the  FSO,  S2,  and  S3.  The  Shared  Event  Analyzer 
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measures  collaboration  (team  performance)  and  force  synchronization. 


Cognitive  Work  Anaiysis  Heat-Map 

As  shown  in  Figure  2,  the  heat  map  function  allows  one  to  view  either  current  'real-time'  work  activity 
(active  nodes)  in  the  0-0  and  D-A  or  alternatively,  summed-up  over  a  specified  epoch  of  time.  Often 
repeated  goals,  plans,  and  beliefs  will  have  the  highest  activation  and  are  depicted  by  the  warmest 
colors.  This  snapshot  of  the  work  domain  is  useful  in  understanding  and  summarizing  work  activity  as  a 
constellation  of  activated  goals  and  plans  to  achieve  them  and  provides  a  detailed  accounting  of 
cognitive  work  activity  across  time.  For  example,  this  tool  could  distinguish  between  two  individuals, 
one  who  sticks  closely  to  an  established  and  circumscribed  work  flow  versus  another  individual  who 
works  dynamically  and  broadly  across  many  different  domain  areas  (i.e.  collaborative  multi-tasking). 


There  is  a  real  need 
for  descriptive  and 
analytical  models  of 
how  cognitive  work 
is  accomplished  with 
both  the  attendant 
real-world 
complexity  and  the 
collaborative 
dimension  of  group 
decision-making. 

There  are  two 
modes  of  running 
the  DAT,  a  real-time 
mode  as  the 
scenario  run  occurs, 
or  a  recall  mode 
after  the  scenario 
run  occurs.  Also,  the 
DAT  can  be  used  by 
multiple  analysts  to 
monitor  multiple 
instances  of  the  WA. 

These  new 

prototype  capabilities  of  an  analyst  to  record  and  view  work  activity  dynamically  and  unobtrusively  (in 
real-time  and  over  specified  epochs)  are,  to  our  knowledge,  unprecedented. 

Cognitive  Workload  Chart 

Cognitive  workload  is  the  amount  of  demand  on  limited  cognitive  resources  required  to  accomplish 
mission  requirements  for  a  human  operator.  The  recognition  of  cognitive  workload  as  an  important 
aspect  of  behavior  emerged  from  studies  of  work  performance  and  subjective  assessments  of  job 
demands  (Wickens  &  Flollands,  2000;  Wierwille  &  Eggemeier,  1993).  Increasing  job  stress  or 
performance  pressure  results  in  performance  improvement  up  to  a  point.  After  that,  increased  job 
demands  sharply  reduce  performance.  This  parabolic  workload-performance  curve  has  been  widely- 


Data  Analysis  Toolset:  Cognitive  Work  Analysis  ‘Heat  Map' 


Figure  2;  Cognitive  work  analysis  'heat  map'  of  Decide-Act  graph  (top)  and  Observe-Orient  graph  (bottom)  activations  as 
the  number  of  times  each  node  was  triggered  across  a  specified  time  period,  here  the  first  seven  minutes  of  the  scenario 
data  run  as  specified  by  the  onset  and  offset  bubbles  of  the  blue  time  slider  bar.  This  tool  can  also  be  used  to  observe 
currently  activated  nodes  if  the  'real  time'  box  is  checked  in  the  top  right.  Warmer  colors  depict  greater  node  activation 
counts;  this  is  user  configurable  where  numerical  ranges  can  be  specified  and  assigned  to  any  selected  color  scheme. 
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established  and  is  known  as  the  Yerkes-Dodson  relationship  (Yerkes  &  Dodson,  1908).  Importantly,  it 
defines  a  zone  of  maximal  performance.  Cognitive  workload  fluctuates  over  time;  in  practice,  this  means 
that  cognitive  workload  is  a  time-series  that  needs  to  be  averaged  over  defined  time  periods  that  are 
short  enough  to  allow  detections  of  low,  optimal,  and  peak  workload  states.  The  goal  of  managing 
Soldier-system 
interactions  is  to  keep 
performance  over  time 
within  the  optimal  region 
of  performance.  Note 
however,  that 
performance  is  variable 
within  an  individual  and 
overtime.  The  optimal 
region  might  fluctuate  due 
to  a  host  of  intra¬ 
individual  factors  such  as: 
fatigue,  multitasking 
demand,  stress,  reduced 
attention  span,  etc. 

Nevertheless,  being  able 
to  objectively  measure 
cognitive  workload  as  a 
time-series  has  practical 
utility  in  better  aligning 
and  achieving  maximal 
Soldier-system 
performance,  as  well  as 
better  understanding 

intra-individual  sources  of  variability.  There  are,  however,  no  reliable,  unobtrusive,  direct  measurements 
of  cognitive  workload.  The  current  standard  of  measuring  cognitive  workload  is  the  NASA  Task  Load 
Index  (see  Hart,  2006),  which  is  a  subjective  survey  method  whose  administration  requires  that  one  stop 
the  field  exercise  or  experiment.  Recent  direct  measurements  of  brain  activity  suggest  a  path  forward, 
but  these  methods  are  in  the  research  stage  and  not  fully  mature  technologies  (see  Dornhege,  Millan, 
Hinterberger,  McFarland,  &  Muller,  2007).  Thus,  currently  unobtrusive  measurement  of  cognitive 
workload  cannot  be  performed  under  realistic  operating  conditions. 

Our  formal  metric  construction  of  cognitive  workload  is  derived  from  the  activation  of  the  user 
workflows  in  the  D-A  graph  that  is  triggered  from  directly  observable  behaviors.  Our  measurement 
approach  provides  an  objective  and  time-series  measure  of  cognitive  workload  that  is  also  defined  by 
meaningful  Mission  Command  behaviors.  As  shown  in  Figure  3,  the  instance  viewer  shows  the  number 
of  concurrently  active  0-0  and  D-A  nodes  that  are  active  either  at  a  given  point  in  time  or  across  a 
specified  time  frame.  This  offers  a  useful  measure  of  cognitive  workload  as  given  by  the  staff  task 
demands  at  any  point  in  time.  In  our  case,  we  define  cognitive  workload  as  the  number  of  concurrently 
active  goals  and  sub-goals  to  achieve  them.  A  limitation  of  the  current  approach  is  that  not  all  goals  are 
equally  complex.  Following  the  approach  of  Bierbaum,  Szabo,  and  Aldrich  (1989)—  who  developed  a 
workload  prediction  model  for  piloting  a  UH-60  helicopter—  it  should  be  possible  to  derive  estimates  of 
the  workload  demands  associated  with  each  goal  node  in  the  D-A  graph;  thus  assigning  to  each  goal  a 
level  of  difficulty  that  can  be  aggregated  into  a  composite  measure  of  cognitive  workload. 


Data  Analysis  Toolset:  Cognitive  Workload 
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Figure  3:  The  activation  of  a  specified  plan,  goal,  or  concept  node  can  be  observed  across  any  period 
of  the  experimental  scenario  run.  In  this  case,  the  goal  node  <Each  Threat  Managed>  has  been 
selected  and  the  dynamics  of  node  activation  and  deactivation  (satisfaction)  are  shown  across  time. 
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Our  metric  is  important  as  it  offers  an  objective  and  continuous  measure  of  cognitive  workload  that  can 
be  collected  unobtrusively.  It  also  has  the  advantage  of  being  part  of  a  system  designed  to  provide 
decision-support.  Node  activations  could  provide  an  unobtrusive  and  continuous  metric  of  cognitive 
workload  that  could  then  be  fed  into  mission  command  systems  as  an  estimate  of  user  workload. 
System  configurations  (interface)  and  capabilities  (alerts)  could  respond  to  this  continuous  estimate  of 
user  cognitive  workload  to  achieve  optimal  Soldier-system  performance. 

Reports  View:  Data  Capture  and  Behavioral  Analysis 


In  scenario-driven 
military 

experiments/training- 
events,  it  can  be 
challenging  to  precisely 
recreate  and  playback 
what  happened  during 
runtime,  even  with  a 
solid  data  collection 
and  analysis  plan.  For 
instance,  it  is  difficult 
to  establish  the 
timeline  of  whether 
and  when  events 
occurred  in  a  scenario- 
driven  military  training 
event,  especially  when 
unplanned 
interventions  and 
schedule  delays  are 
introduced  during 
execution  time. 

Precisely  capturing  the 
timing  of  events  and 
information  flows  is 

critical  to  enabling  the  military  analytical  community  to  advance  a  scientific  capability  to  understand  and 
address  the  challenges  and  complexities  inherent  in  network-enabled  mission  command. 

The  Data  Analysis  Tool  addresses  these  challenges  by  capturing  D-A  and  0-0  graph  instance  start  and 
stop  times,  monitor  events,  and  notification  events  for  each  running  instance  of  the  Warfighter 
Associate.  This  is  essential  for  defining  performance.  The  Reports  View  is  shown  in  Figure  4.  With  this 
tool  the  analyst  can  search  the  database  generated  by  the  Warfighter  Associate  system  in  order  to 
better  understand  Soldier  and  system  performance.  The  report  can  be  filtered  and  sorted  on  number  of 
dimensions  (see  Figure  4,  right  panel)  to  include:  associate,  start  time,  update/stop  time,  duration, 
name,  attributes,  type,  and  lifecycle  state. 

Based  on  the  Reports  View,  key  performance  metrics  include:  operational  tempo,  resource 
management,  and  information  flows.  Performance  can  be  defined  in  terms  of  operational  tempo  (task 
completion  time),  efficiency  of  resource  allocation,  tracking  when  battlefield  events  actually  occurred 
during  scenario  run-time  and  when  and  what  recommendations  were  given  to  the  users.  Operational 


Data  Analysis  Toolset:  Log  Analyzer  and  Task  Completion  Timing 


Figure  4:  The  log  analyzer  catalogs  the  Decide-Act  (D-A)  graph  Instance  start  and  stop  times, 
monitor  events,  and  notification  events  for  each  running  instance  of  the  Warfighter  Associate.  The 
log  files  generated  by  the  system  can  be  filtered  (right  panel)  in  order  to  better  understand 
Soldier-system  performance  and  extract  metrics  of  task  completion  timing,  such  as  the  onset  and 
duration  of  critical  battlefield  events,  resource  allocation,  and  the  timeline  of  information  pushed 
by  the  Soldiers  and  the  associate  system. 
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tempo  refers  to  how  quickly  the  staff  is  able  to  cycle  through  decision-making  in  effecting  the 
battlefield.  The  Timeline  Report  provides  task  completion  timing,  the  onset  and  duration  (to  goal 
satisfaction)  of  critical  battlefield  events.  Resource  management  refers  to  how  well  the  staff  is  able  to 
coordinate  and  prioritize  the  use  of  limited  battlefield  resources.  This  is  especially  the  case  with 
Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  assets.  The  Warfighter  Associate  monitors  the 
location  and  status  of  all  assets  in  the  battle-space  and  pushes  recommendations;  and  all  of  this  data  is 
captured  in  log  files  and  a  database.  Using  the  Reports  View,  an  analyst  can  determine  whether  and 
when  assets  were  tasked  to  assess  efficiencies.  Finally,  the  Log  Analyzer  gives  access  to  the  full  timeline 
of  information  as  it  is  reported  or  pushed  on  the  network  (i.e.  chat  room  or  voice  communication 
transcribed  to  text)  by  the  Soldiers  and  the  Warfighter  Associate  system.  This  is  essential  for  capturing 
and  analyzing  the  complex  information  flows  that  occur  in  the  Mission  Command  network. 

Shared  Event  Analyzer 

A  shared  event  is  defined  as  activated  0-0  and  D-A  nodes  that  are  shared  by  multiple  WA  Instances,  in 
our  case  by  any  combination  of  the  S2  (Intelligence  Officer),  S3  (Operations  Officer),  and  FSO  (Fire 
Support  Officer).  The  Sharing  Report  captures  the  time  and  content  of  each  shared  0-0  and  D-A  node 
instance  and  displays  a  representation  of  the  shared  events  between  multiple  WA  Instances.  This  is 
illustrated  in  Figure  5  whereby  the  color  of  the  D-A  nodes  (for  the  MEDEVAC  hierarchical  sub-tree) 
denotes  unique  or  shared  responsibilities.  In  this  example,  task  responsibilities  that  are  shared  by  all 
three  role  positions  (S3,  S2,  FSO)  are  shown  in  grey,  whereas  those  shared  by  the  S3  and  FSO  in  orange, 
and  those  unique  to  the  S3  or  FSO  are  in  green  and  yellow,  respectively.  Two  metrics  can  be  derived 
from  such  a  representation:  collaborative  information  flows  and  force  synchronization. 

Collaborative  Information  Flows 

Most  tasks  are  not  planned  or  executed  in  isolation,  either  in  terms  of  other  tasks  or  in  terms  of  other 
actors  and  stakeholders.  The  essence  of  teamwork  is  the  ability  to  efficiently  maintain  a  coherent  set  of 
tasks  across  many  actors  and  assets  in  a  manner  that  does  not  conflict  with  one  another.  Geddes  (1994) 
provides  an  analysis  of  conflicts  across  multiple  actors  and  assets  to  identify  two  separate  families  of 
conflict:  Goal  Conflicts  and  Plan  (or  activity)  Conflicts.  Recognizing  that  both  goals  and  activities  are 
hierarchical  in  nature,  this  analysis  explicitly  addresses  sub-goals  and  sub-activities  and  the  differences 
between  purposeful  state  changes  (goals)  and  incidental  state  changes  ("side  effects")  at  different  levels 
of  activity  aggregation  and  decomposition.  As  noted  by  the  author,  two  plans  that  may  seem  perfectly 
compatible  at  one  level  of  aggregation  may  have  serious  side  effect  conflicts  at  a  lower  level  of 
decomposition  that  blocks  their  successful  joint  execution  across  the  team.  Geddes  also  makes  the  case 
that  only  some  of  the  conflicts  can  be  detected  and  avoided  at  "design  time."  From  a  systems 
engineering  and  behavioral  research  perspective,  there  is  always  a  fundamental  need  for  "execution 
time"  conflict  detection  and  correction  across  teams  (Geddes,  2011). Our  method  of  tagging 
collaborative  information  flows  and  the  metric  of  force  synchronization  are  tools  that  the  analyst  can 
use  to  assess  teamwork  during  execution  time. 

The  method  of  tagging  collaborative  information  flows  on  the  D-A  graph  (as  input  and  output  arrows) 
allows  one  to  determine  whether  these  staff  processes  were  mutually  supported  during  task  execution. 
A  sub-section  of  the  D-A  is  illustrated  in  Figure  5.  In  this  illustrated  sub-section,  the  plan  [Use  Air 
MEDEVAC]  is  tagged  with  arrows  where  necessary  collaborative  information  flows  are  represented.  The 
FSO  is  responsible  for  responding  to  an  air  MEDEVAC  request.  The  Warfighter  Associate  reads  the 
MEDEVAC  request  from  the  chat-room  input  and  then  prompts  the  FSO  by  posting  this  request  on  their 
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graphical  user  interface.  The  arrow  to  the  right  of  the  [Use  Air  MEDEVAC]  plan  indicates  that  the  WA 
shares  this  information  with  both  the  S3  and  S2.  The  operations  officer,  S3,  is  responsible  for  all  force 
protection  activities  of  Soldiers  and  civilians  in  location  of  the  designated  MEDEVAC  site.  And  if  the 
MEDEVAC  was  induced  by  enemy  activity,  then  it  is  important  that  the  intelligence  officer,  the  S2, 
provide  actionable  intelligence  to  the  MEDEVAC  crew  on  the  enemy  disposition  in  the  area. 

The  illustrated  sub-tree  details  mission  concepts  and  staff  workflows,  such  as  designating  and  providing 
ground  security  for  the  recovery  site  as  well  as  air  security  for  the  MEDEVAC.  In  satisfying  the  sub-goal 
<Ground  Security  Provided  for  Site>  the  plan  [Task  Unit  to  Secure  Site]  is  activated  if  a  unit  is  tasked  to 
provide  site  security.  The  arrows  indicate  a  necessary  collaboration  between  the  FSO  managing  the 
MEDEVAC  request  and  the  S3,  responsible  for  establishing  positive  control  over  the  designated  recovery 


Figure  5:  Illustrated  Sub-section  of  Decide-Act  (D-A)  Graph  Hierarchy  for  MEDEVAC  Operations 


site  by  cordoning  off  the  area.  In  tagging  the  D-A  with  collaborative  information  flows,  the  Warfighter 
Associate  actively  supports  collaboration.  The  analyst  uses  the  Reports  View  toolset  to  determine 
whether  mission  concepts  and  staff  workflows  were  indeed  supported. 

Force  Synchronization 


The  metric  of  force  synchronization  attempts  to  measure  collaborators'  coordination  of  shared  activities 
and  purpose  across  time  and  space.  The  concept  of  force  synchronization  applies  to  groups  of  actors 
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that  desire  to  arrive  at  a  set  of  goal  states  by  achieving  a  set  of  intermediate  states  in  a  specific 
sequence  (Geddes,  2011).  While  the  sequence  of  intermediate  states  is  often  described  in  terms  of  time 
and  space,  force  synchronization  can  also  refer  to  the  synchronization  of  staff  workflows  along  the  social 
and  cognitive  dimensions.  For  example,  in  the  case  of  Soldier-information  systems,  synchronization  may 
involve  communication  and  decision  sequences  with  accompanying  states  of  data  and  awareness. 
Synchronization  exists  by  degree.  Perfect  synchronization  can  be  said  to  exist  when  all  of  the  actors 
achieve  each  of  the  intermediate  states  they  are  responsible  for  according  to  the  expectations  of  the 
other  actors  and  in  a  manner  that  does  not  create  a  conflict,  either  direct  or  indirect,  between  the  actors 
(Geddes,  2011).And,  synchronization  is  an  integrative  measure  that  occurs  over  a  period  of  time  rather 
than  at  an  instant.  As  a  result,  there  is  a  characteristic  time  period  over  which  synchronization  can  be 
measured.  The  time  period  chosen  for  measurement  should  be  long  enough  to  capture  the  full  time- 
series  of  shared  activity. 

Our  metric  definition  of  force  synchronization  is  based  on  the  timely  co-activation  of  essential  tagged  D- 
A  nodes  by  collaborators  in  response  to  shared  significant  events  (e.g.  battle  drills)  across  a  period  of 
time.  For  instance,  the  timely  response  and  satisfaction  of  an  air  MEDEVAC  request  requires  that  the 
staff  quickly  work  through  several  collaborative  sequences  in  tandem  to  satisfy  several  intermediate 
goal  states,  from  designating  and  tasking  units  to  secure  the  recovery  site  (S3),  to  establishing  an  air 
corridor  and  air  security  (FSO),  to  providing  intelligence  on  enemy  force  disposition  in  the  immediate 
vicinity  and  threats  along  the  air  corridor  (S2).A  staff  with  a  high-degree  of  force  synchronization  can 
achieve  their  intermediate  goal  states  in  a  timely  manner  that  does  not  create  conflict  with  one  another. 
The  Data  Analysis  Toolset  allows  the  analyst  to  view  the  activation  dynamics  of  the  D-A  graph  and 
possibly  identify  friction  points  of  collaboration  that  contributed  to  poor  force  synchronization,  or  simply 
assumptions  and  workflows  that  were  overlooked.  For  instance,  perhaps  an  inordinate  amount  of  time 
was  spent  satisfying  the  goal  <Air  Security  Provided  for  MEDEVAC>,  or  alternatively,  perhaps  the  goal 
was  not  attended  to,  nor  satisfied.  The  analyst  would  then  be  clued  in  to  determine  why  that  was 
indeed  the  case.  Possible  factors  influencing  force  synchronization  include  the  quality  of  task  execution, 
situation  awareness,  shared  understanding,  and  team  collaboration.  Continuing  our  example,  perhaps 
the  staff  had  an  incomplete  picture  of  enemy  air  threats  and  thus  spent  a  considerable  amount  of  time 
seeking  information  and  then  deliberating  whether  to  request  additional  air  security  assets  (e.g.  Apache 
helicopters).  Or  alternatively,  perhaps  the  staff  did  not  challenge  their  assumptions  and  implicitly 
assumed  the  mental  frame  of  no  known  enemy  air  threats.  In  sum,  the  activation  dynamics  of  the  D-A 
nodes  provides  the  analyst  with  a  state-trace  of  the  collaborative  workflows  underlying  our  proposed 
metric  of  force  synchronization  and  important  markers  to  uncover  the  governing  dynamics  of  staff 
collaboration. 


VI.  METHODS  &  EXPERIMENTAL  SCENARIO 

Our  humans-in-the-loop  experiment  was  conducted  in  our  laboratory  facility  at  Aberdeen  Proving  Ground 
and  featured  a  finely  detailed,  high  fidelity  scenario  scripting  the  actions  of  four  battalions  as  well  as  host 
nation  military  and  police  units  acting  within  an  area  of  interest  to  a  larger,  parent  military  unit,  a  brigade. 
The  level  of  analysis  and  agency  in  the  experiment  scenario  was  the  brigade  which  included  three  soldier 
participants  led  by  a  senior  non-commissioned  officer  (NCO)  and  two  lower  ranked  NCOs.  The  senior 
NCO,  a  Staff  Sergeant  (E-6)  with  10  years  experience,  assumed  the  role  of  an  operations  officer  (S3).  The 
other  two  NCOs  were  Sergeants  (E-5)  with  3  and  4  years  experience  who  assumed  the  role  of  an 
intelligence  officer  (S2)  and  a  fire  support  officer  (FSO),  respectively.  The  NCOs  all  experienced  at  least 
one  combat  tour  under  counter  insurgency  (COIN)  operations  in  either  Iraq  or  Afghanistan  and  were 
accomplished  in  their  role-position.  These  soldiers  were  from  the  same  organic  unit,  understood  their 
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role  positions,  and  performed  well  and  cohesively  within  the  combat  vignettes  of  the  scenario. 


The  experiment  was  conducted  as  a  two-day  event.  The  first  day  was  devoted  to  reading  the  soldier 
participants  into  the  scenario,  which  included  a  "road  to  war"  briefing  by  the  scenario  co-author  and 
officer-in-charge  (Lieutenant  Colonel,  0-5).  Soldiers  were  given  a  full  dossier  of  information  including 
intelligence  summaries,  preplanned  fires,  ISR  synchronization  matrices,  and  key  coordinating  maneuver 
graphics  supporting  their  upcoming  responsibilities.  All  three  soldiers  were  knowledgeable  of  the  mission 
command  system  used  to  maintain  a  common  operational  picture  during  the  experiment,  the  Command 
Post  of  the  Future  (CPOF).  Soldiers  received  three  hours  of  training  using  CPOF  to  be  assured  of  a  base 
level  of  soldier  performance  using  the  system.  The  second  day  consisted  of  the  four  hour  experimental 
run  (interspersed  with  three  20  minute  breaks  or  pauses)  followed  by  an  after-action  review  conducted 
by  the  officer-in-charge.  The  experimental  scenario  consisted  of  COIN  operations  and  key  events/threads 
included:  (a)  a  host  nation  convoy  transporting  raw  "uranite"  fissile  material  through  their  area  of 
operations,  (b)  finding,  fixing  and  conducting  a  cordon  and  search  operation  versus  a  high  value  target 
(FIVT),  (c)  a  planned  enemy  attack  on  host-nation  parade  dignitaries  using  an  improvised  explosive  device 
(lED),  (d)  responding  to  enemy  mortar  attacks,  (e)  supervising  the  repositioning  of  a  critical  radar  asset,  (f) 
missing  ammunition,  and  (g)  recovery  of  a  downed  civilian  aircraft  in  a  hostile  border  area. 

Since  the  action  took  place  within  a  brigade  area  of  interest,  the  soldiers  assumed  responsibilities  for  the 
operation  of  a  brigade  tactical  operation  center  (TOC).  They  performed  several  functions.  First,  they 
monitored  the  battlefield  environment  and  Mission  Command  inputs  via  six  chat  rooms.  Four  of  the  chat 
rooms  were  organized  by  Mission  Command  function  (maneuver,  intelligence,  fires,  and  sustainment), 
one  represented  Joint  Special  Operations  Command  and  higher  echelons  of  command  (JSOC/FIICON),  and 
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FiaureO:  Timeline  of  voice  and  chat  room  communication  activity  by  the  Soldier  participants —  intelligence  (S2),  maneuver 
(S3),  and  fire  support  officer  (FSO) —  across  the  experiment  sorted  by  scenario  threads.  Scenario-driven  chat  room  inputs 
of  significant  battlefield  information  are  depicted  as  yellow  ticks.  Experimental  pauses  are  shown  in  gray. 


16 


another  simulated  liaison  lash-ups  with  host  nation  units  (Gorgasnet).  Gorgas  was  the  name  given  to  the 
host  nation.  The  experimental  scenario  (Buchler  &  Moyers,  2012)  consisted  of  three  hours  of 
approximatelySOO  scripted  chat  room  inputs  from  over  40  scripted  actors  representing  both  routine  chat 
room  messages  and  those  in  response  to  scenario  events.  The  experiment  itself  lasted  four  hours 
because  of  the  three  20  minute  breaks  added  during  runtime. 

Second,  the  soldiers  processed  all  of  the  incoming  chat  room  information  and  used  CPOF  to  maintain  a 
common  operational  picture  (COP)  of  significant  events  occurring  within  their  area  of  operations.  Third, 
the  soldiers  interacted  with  the  synthetic  (i.e.  scripted)  actors  in  the  chat  rooms,  for  instance  by 
requesting  additional  information,  tasking  subordinate  battalion  units,  re-tasking  ISR  assets,  coordinating 
with  the  host  nation  units,  or  issuing  regular  reports  to  higher  echelons  of  command.  Responses  to 
requests  for  information  were  handled  by  an  experimental  team,  the  "white  cell",  who  inserted  inputs 
into  the  various  chat  rooms  assuming  the  names  of  the  various  synthetic  actors.  Friendly  unit  locations 
were  displayed  on  the  COP  to  simulate  the  Mission  Command  functionality  of  Blue  Force  Tracker.  The 
white  cell  also  handled  the  movement  of  friendly  unit  icons  around  the  COP  to  include  the  re-tasking  of 
unit  and  ISR  assets. 

The  Soldier  participants  each  sat  in  separate  sound  attenuated  experimental  chamber  in  front  of  two 
screens,  one  populated  with  the  inputs  to  the  six  chat  rooms  and  the  other  with  the  CPOF  Mission 
Command  System  that  functioned  as  the  brigade  COP.  The  isolation  from  one  another  forced  the 
participants  to  communicate  via  voice  (headset)  or  the  chat  room  interface.  Voice  communications  were 
in  a  common  channel. 

The  data  recorded  during  the  scenario  experiment  consisted  of  a  log  of  all  voice  and  chat  room 
communications,  screen  video  recordings,  and  the  log  output  of  Warfighter  Associate.  The  three 
instances  of  the  Warfighter  Associate,  one  for  each  duty  position  S2-S3-FSO,  were  run  in  the  background. 
This  means  that  decision  aiding  was  turned  off  and  the  recommendations  generated  by  the  system  were 
not  provided  to  the  soldier  participants.  Flowever,  the  Warfighter  Associate  did  collect  data  during  the 
experimental  scenario,  which  was  fed  into  the  data  analysis  tool  (DAT).  During  the  experimental  scenario, 
the  white  cell  could  observe  in  real  time  the  metrics  described  in  the  next  section,  our  results  (see  Figures 
7  and  8). 

VII.  RESULTS 

Timeline  of  Staff  Communications 

The  experimental  scenario  proved  to  be  quite  challenging  to  the  Soldier  participants,  requiring  a  high- 
degree  of  staff  collaboration.  Figure  6  provides  a  raw  timeline  of  staff  voice  and  chat  room 
communications,  sorted  by  experimental  scenario  thread  and  corresponding  ticks  of  significant  chat  room 
inputs.  The  significant  chat  room  inputs  were  often  reports  relayed  into  chat  rooms  by  battalion 
operators  in  contact  with  subordinate  units  at  the  tactical  level  or  collection  managers  of  ISR  assets.  It  is 
evident  that  the  Soldier  participants  were  challenged  with  dealing  with  multiple  overlapping  events.  For 
instance,  the  communication  pattern  of  the  maneuver  (S3)  NCO  suggests  that  he  was  able  to  successfully 
manage  multiple  communications  threads  about  multiple  threats  as  they  occurred.  In  addition,  the  data 
also  suggests  that  the  maneuver  (S3)  NCO  sustained  attention  to  particular  scenario  threads  such  as  the 
uranite  convoy  moving  through  the  area  of  operations,  perhaps  indicating  command  relevance  and 
prioritization  of  effort.  Next,  we  used  the  DAT  to  examine  the  activation  data  provided  by  the  WA  system. 
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Cognitive 
Workload  Results 

The  Data  Analysis 
Toolset  facilitates 
the  analysis  of  the 
state  traces  of 
node  activations 
captured  by  the 
WA  system,  which 
we  use  to  develop 
a  novel  metric  for 
cognitive  workload 
to  assess  Mission 
Command 
performance.  For 
the  analyst,  the 
utility  of  new 
metrics  are 
determined  by 
how  well  they 
capture  meaningful 

staff  performance.  This  depends  on  how  well  the  WA  system  represents  activated  domain  knowledge 
over  the  course  of  execution  time.  These  results  serve  as  a  proof  of  concept  and  a  concrete  example  of 
the  utility  of  our  metric  framework  and  analytical  approach  using  the  WA  system. 

In  Figure  7,  the  activation  count  of  one  goal  node  from  the  D-A  graph  is  depicted  across  the  experimental 
scenario.  This  high-level  goal  node  <Each  Threat  Managed>  tracks  current  threats  in  the  area  of 
operations.  As  a  metric,  the  activation  profile  of  this  high-level  goal  node  represents  the  many  different 
workflows  of  responding  to  threats.  The  activation  count  increases  steadily  as  more  and  more 
compounding  events  occur  in  the  area  of  operations.  In  representing  the  number  of  active  workflows, 
the<Each  Threat  Managed>  goal  node  is  perhaps  a  good  candidate  for  a  measure  of  cognitive  workload 
for  the  staff.  Deactivations  or  satisfaction  of  particular  instances  of  the  goal  node  <Each  Threat 
Managed>  are  depicted  as  well  and  occur  when  threats  are  successfully  managed  in  the  area  of 
operations.  The  DAT  can  be  consulted  for  a  more  precise  accounting  of  exactly  which  threat  was 
successfully  managed  at  a  particular  time-step. 

ISR  Resource  Management  Results 

Managing  and  prioritizing  limited  ISR  resources  is  an  important  Mission  Command  function.  Flere  we 
examine  the  activation  count  of  a  goal  node  from  the  D-A  graph  <Threat  Surveillance  Adeguate>  that  is 
activated  when  an  active  threat  exists  in  the  area  of  operations  and  there  is  no  ISR  asset  in  range  to  cover 
the  threat.  The  WA  system  monitors  the  current  location  of  all  known  ISR  assets  and  is  knowledgeable  of 
their  capabilities.  The  <Threat  Surveillance  Adeguate>goa\  is  satisfied  when  an  ISR  asset  moves  in  range  to 
cover  the  threat,  which  typically  requires  that  the  staff  task  or  re-task  ISR  assets.  In  Figure  8,  the 
activation  count  of  the  <Threat  Surveillance  Adeguate>  goal  is  depicted  across  our  experimental  scenario. 
Note  that  for  the  first  half  of  the  experimental  scenario  (until  around  the  1230  time-step)  the  <Threat 
Surveillance  Adeguate>  goal  is  efficiently  satisfied,  staying  at  zero.  Afterward,  the  staff  is  unable  to  satisfy 
all  the  surveillance  demands  in  the  area  of  operations  given  limited  ISR  resources. 

Manifest  Agility  Results 

The  time  course  of  goal  activation  can  also  be  rendered  against  a  standard  baseline  of  comparison.  The 
dotted  lines  in  Figure  8  reflect  the  time  course  of  ISR  needs  across  the  experimental  scenario  with  no  staff 


Timeline  of  Activation  of  the  Goal  Node  <Each  Threat  Managed> 
Across  the  Experimental  Scenario 


Figure?:  Timeline  of  the  activation  and  deactivation  (satisfaction)  counts  for  a  specific  goal  node  <Each  Threat  Managed> 
depicted  across  the  entire  experimental  scenario.  The  <Each  Threat  Managed>  goal  node  is  perhaps  a  good  candidate  for  a 
measure  of  cognitive  workload  for  the  whole  Mission  Command  staff. 
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inputs  meaning  that 
the  ISR  assets 
remained  static  with 
no  tasking  or  re¬ 
tasking.  To  arrive  at 
this  graph,  the 
experimental 
scenario  is  run  in  a 
baseline  condition— 
a  dry  run  without 
staff  involvement- 
represented  by  the 
dotted  line.  The 
experimental 
condition  is  then  run 
with  staff 

involvement  given  by 
the  red  line  in  Figure 
8.  This  difference  is  a 
measure  of  what  has 

been  termed  manifest  agility  by  the  Command  and  Control  (C2)  operational  research  community 
(Alberts,  2011).  According  to  Alberts  and  Tillman  (2012),  manifest  agility  is  defined  by  the  response  to  an 
event  that  has  occurred  and  this  involves  comparing  what  actually  happened  to  what  would  have 
happened  if  no  change  had  taken  place.  Thus,  the  goal  activation  dynamics  depicted  in  Figure  8  reflect 
the  manifest  agility  of  the  staff  and  in  effecting  ISR  management. 


Timeline  of  Activation  of  the  Goal  Node  <Threat  Surveillance  Adequate> 
Across  the  Experimental  Scenario 


Figures:  Timeline  of  the  activation  and  deactivation  (satisfaction)  counts  for  a  specific  goal  node  <Threat  Surveillance 
Adequate>  depicted  across  the  entire  experimental  scenario.  The  <Threat  Surveillance  Adequate>  goal  node  is  triggered 
when  and  there  are  no  ISR  assets  in  range  of  an  existing  threat  in  the  area  of  operations.  This  is  a  measure  of  the  allocation 
of  ISR  resources,  which  are  perhaps  limited  in  relation  to  the  number  of  threats  during  the  latter  half  of  the  scenario.  The 
gray  dotted-line  represents  a  baseline  dry  run  condition  with  no  Soldier  participation  where  ISR  assets  have  not  been 
tasked  or  re-tasked  in  response  to  scenario  events. 


VIII.  CONCLUSIONS 

Objectively  measuring  Mission  Command  effectiveness  is  a  formidable  Army  challenge  given  the  difficulty 
capturing  the  intricacies  of  individual  and  team  cognition,  particularly  as  Soldiers  confront  environmental 
challenges  with  multiple  overlapping  problem  spaces.  The  WA  system  provides  one  possible  metric 
solution  by  automating  collection  and  the  aggregation  of  empirical  data  derived  from  field  or  laboratory 
studies.  The  WA  provides  capabilities  to  improve  and  evaluate  performance  in  Mission  Command, 
including  difficult  to  quantify  concepts  such  as  cognitive  workload,  resource  management,  and  agility. 
Importantly,  the  WA  also  provides  a  computational  instantiation  of  Soldier  workflows,  represented  in  the 
0-0  and  D-A  graphs,  from  which  to  develop  predictive  performance-based  models  of  emergent  problem 
solving  behaviors  in  complex  collaborative  environments. 

An  important  future  direction  for  this  research  is  to  validate  these  measures  of  cognitive  workload  and 
force  synchronization  in  additional  laboratory  experiments  and  to  scale-up  this  effort  to  capture  data 
from  military  training  events  as  field  studies.  The  cognitive  workload  measure,  for  instance,  could  be 
compared  to  standard  subjective  measures,  such  as  the  point  estimates  of  the  NASA  Task  Load  Index 
(Flart,  2006),  as  well  as  more  direct  measurements  of  brain  activity.  In  the  latter  case, 
electroencephalography  can  be  used  to  record  and  render  a  time  series  of  electrical  activity  along  the 
scalp.  It  would  be  very  interesting  to  examine  whether  there  exists  a  robust  neural  signature  associated 
with  the  time  course  of  goal  activation,  as  inferred  from  the  state  traces  of  the  WA. 

Our  metric  approach  can  also  be  used  to  assess  the  effectiveness  of  any  prototype  technology.  In  this 
case  the  experimental  design  requires  two  conditions,  one  with  and  another  without  the  prototype 
technology.  Used  in  this  way,  the  WA  would  simply  run  in  the  background  and  not  contribute  decision- 
support.  Instead,  the  WA  would  provide  the  metrics  framework  for  assessing  staff  performance  with  and 
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without  the  prototype  technology.  For  the  acquisition  community,  these  metrics  could  provide  solid 
benchmarks  of  performance  to  assess  the  effectiveness  of  any  prototype  technologies,  such  as  a  new 
mission  command  system,  across  the  development  cycle  on  a  range  of  dimensional  metric  parameters. 

One  potential  application  of  the  WA  is  training  technology.  The  alerts  and  decision  aids  offers  Soldier's 
support  in  Mission  Command  and  performance  can  also  be  evaluated  and  used  during  debriefings  and 
after-action  reviews.  The  WA  could  reduce  training  time,  and  thus  training  costs,  while  increasing  Mission 
Command  performance  through  situated  practice  in  scenario-based  training  and  in  the  application  of 
doctrinal  knowledge.  In  terms  of  practical  utility,  the  WA  also  potentially  addresses  organizational  and 
environmental  challenges  in  providing  decision-aiding  to  Soldiers  whom  are  inexperienced,  fatigued,  or 
both. 

In  sum,  the  WA  addresses  a  major  tenet  of  the  U.S.  Office  of  Secretary  of  Defense's  "data  to  decisions" 
initiative  (Swan  &  Hennig,  2012)  and  the  primary  challenge  for  military  commanders  and  their  staff  to 
shorten  the  cycle  time  from  data  gathering  to  decisions.  The  WA  provides  a  novel  class  of  metrics  to 
assess  the  operational  efficiency  of  mission  command.  Our  approach  may  be  an  important  development 
toward  the  goal  of  a  mature  scientific  experimental  infrastructure  and  the  measures  to  capture  Mission 
Command  effectiveness,  focused  on  theory  development,  experimentation,  hypothesis  testing,  and  data 
collection  relevant  to  operations. 


IX.  REFERENCES 

Alberts,  D.  S.  (2011).  The  Agility  Advantage:  A  survival  guide  for  complex  enterprises  and  endeavors.  DoD 
Command  and  Control  Research  Program. 

Alberts,  D.,  &  Garstka,  J.  Network  Centric  Operations  Conceptual  Framework  Version  2.0;  Evidence  Based 
Research,  Inc.:  Vienna,  VA,  2004. 

Alberts  D.,  &  Tillman  M.  (2012,  January)  NEC2  Effectiveness  and  Agility:  Analysis  Methodology,  Metrics, 
and  Experimental  Results.  Conference  Presentation  at  the  Military  Operational  Research  Society  (MORS) 
Workshop:  Joint  Framework  for  Measuring  C2  Effectiveness.  Laurel,  MD. 

Akerkar,  R.  and  Sajja,  P.  (2011)  Knowledge-Based  Systems.  Jones  and  Bartlett  Publishers:  Sudbury,  MA. 

Atkinson,  S.,  &  Moffat,  J.  (2007)  The  Agile  Orgon/zof/on.  Washington  D.C.:  Command  and  Control  Research 
Program  (CCRP)  Publication  Series. 

Bierbaum,  C.,  Szabo,  S.  &  Aldrich,  T.  (1989)  Task  analysis  of  the  UFI-60  mission  and  decision  rules  for 
Developing  a  UFI-60  Workload  Prediction  Model.  Army  Research  Institute,  Research  Product  89-08. 

Boyd,  J.  (1986)  "Organic  Design  for  Command  and  Control,"  in  "Discourses  on  Winning  and  Losing," 
(Unpublished  slide  presentation  archived  at  the  Marine  Corps  University  Research  Archives,  Quantico, 
VA),  charts  16,  25. 

Buchler,  N.  &  Moyers,  R.  (2012,  in  preparation)  A  Military  Scenario  for  Brigade-level  Counter  Insurgency 
(COIN)  Operations.  ARL-Technical  Report  Series 

National  Research  Council,  Committee  on  Network  Science  for  Future  Army  Applications  (CNSFAA), 
Network  Science.  The  National  Academies  Press.  Washington,  DC,  2005. 

Dornhege,  G.,  Millan  J.,  Flinterberger,  T.,  McFarland,  D.  &  Muller,  K-R.  (Eds)  (2007)  Toward  Brain- 
computer  Interfacing.  Massachusettes  Institute  of  Technology 

Endsley,  M.  &  Garland,  D.  (2009)  Situation  Awareness:  Analysis  and  Measurement.  Mahwah,  NJ: 
Lawrence  Erlbaum. 

FichtI,  Ted  COL  (ret.).  Ft.  Leavenworth,  KS.  Personal  communication.  May  10,  2010 


20 


Geddes,  N.  (1994)  A  Model  for  Intent  Interpretation  for  Multiple  Agents  with  Conflicts.  Alpharetta,  GA: 
Veloxiti,  Inc. 

Geddes,  N.  (2011)  Measures  of  Performance  for  Collaborating  Mission  Command  Teams.  Alpharetta, 

GA:  Veloxiti,  Inc. 

Hart,  S.  (2006)  NASA-Task  Load  Index  (NASA-TLX);  20  Years  Later.  Human  Factors  and  Ergonomics  Society 
Annual  Meeting  Proceedings,  50,  904-908 

Klein,  G.,  Orasanu,  J.,  Calderwood,  R.  &  Zsambok,  C.  (Eds.)  (1993).  Decision  making  in  action:  Modeis  and 
methods.  Norwood,  NJ:  Ablex  Publishing  Corporation 

Lynch,  K.R.  (2008)  Army  digital  systems  complexity.  DTIC  Document,  pp.  1-48. 

Parasuraman,  R.,  Sheridan,  T.B.,  and  Wickens,  C.D.  (2000)  A  model  for  types  and  levels  of  human 
interaction  with  automation.  IEEE  Transactions  on  Systems,  Man,  and  Cybernetics-Part  A:  Systems  and 
Humans,  30(3),  286-297. 

Parasuraman,  R.,  and  Riley,  V.  (1997)  Humans  and  automation:  Use,  misuse,  disuse,  abuse.  Human 
Factors,  39(2),  230-253. 

Polk,  P.  (1999).  A  critique  of  the  Boyd  Theory:  Is  it  relevant  to  the  Army?  Fort  Leavenworth,  KS: 

Command  and  General  Staff  College. 

Swan,  J.  and  Hennig,  J.,  From  Data  to  Decisions.  Army  AL&T  (Acquisition,  Logistics  &  Technology), 
January-March  2012. 

Walker,  G.,  Stanton,  N.,  Salmon,  P  &  Jenkins  D.  (2009)  Command  and  Control:  The  Sociotechnical 
Perspective.  Burlington,  VT:  Ashgate. 

Wickens,  C.,  &  Hollands,  J.  (2000)  Engineering  Psychology  and  Human  Performance,  3rd  ed.  Upper 
Saddle  River,  NJ:  Prentice-Hall. 

Wierwille,  W.  and  Eggemeier,  F.  (1993)  "Recommendations  for  Mental  Workload  Measurement  in  a  Test 
and  Evaluation  Environment,"  Human  Factors,  vol.  35,  pp.  263-281. 

Yerkes,  R.  &  Dodson,  J.  (1908)  "The  Relationship  of  Strength  of  Stimulus  to  Rapidity  of  Habit  Formation. 
Journal  of  Comparative  Neurology  and  Psychology,  18,  459-482. 


21 


